Add new `--vid-stride` inference parameter for videos #9256

VELCpro · 2022-09-02T09:42:29Z

@glenn-jocher as seen in the issue #8079 (comment) there's the need sometimes to skip frames while performing inference on a video.
One of the useful use cases could be batch processing of very long archive videos where high frame rates are not needed.

I added two parameters to detect.py

--inf-rate --> number of inference given a frame rate unit
--fr-unit --> frame rate unit ('fps', 'fpm', 'fph')

Where :

fps = frames per second
fpm = frames per minute
fph = frames per hour

example use:
python detect.py --device 0 --weights yolov5s.pt --inf-rate 1 --fr-unit fps --source vid.mp4

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancement to video frame processing with the introduction of frame-rate stride control.

📊 Key Changes

🖼️ Added a vid_stride parameter to control the stride of video frame processing (skipping frames).
🛠️ vid_stride is implemented in LoadImages and LoadStreams classes, affecting how videos are loaded and processed.
🎛️ The vid_stride option is added as a command-line argument in both predict.py and detect.py scripts, permitting customization from the command-line interface.

🎯 Purpose & Impact

🚀 The purpose is to allow users to speed up video processing by analyzing fewer frames (skipping over frames based on stride value).
✨ This can significantly reduce processing time and resource usage when high frame-rate precision is not required.
🌐 It can also be beneficial for streaming scenarios where live feedback is more important than processing every single frame.

github-actions

👋 Hello @VELCpro, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

✅ Verify your PR is up-to-date with ultralytics/yolov5 master branch. If your PR is behind you can update your code by clicking the 'Update branch' button or by running git pull and git merge master locally.
✅ Verify all YOLOv5 Continuous Integration (CI) checks are passing.
✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

glenn-jocher · 2022-09-02T11:51:01Z

@VELCpro thanks for the PR! The code additions are too complicated to be added to master, especially the multiple units, but there is an example in LoadStreams with a single variable called read that manages frame skipping that we could use as a starting point.

A couple things to remember here are that detect.py and classify/predict.py would both need to be updated, and that LoadStreams and LoadImages should share common arguments and funcitonalities, so if we exposed a new variable at the argparser it should affect both identically, i.e. --variable 30 to read at 1 frame out of every 30, default --variable 1

yolov5/utils/dataloaders.py

Lines 377 to 383 in 9da6d0f

    
           n, f, read = 0, self.frames[i], 1  # frame number, frame array, inference every 'read' frame 
        
           while cap.isOpened() and n < f: 
        
               n += 1 
        
               # _, self.imgs[index] = cap.read() 
        
               cap.grab() 
        
               if n % read == 0: 
        
                   success, im = cap.retrieve()

VELCpro · 2022-09-02T12:41:12Z

@glenn-jocher thanks for the feedback!
I can try to simplify the code using only a sort of "skip_frame" variable

for more information, see https://pre-commit.ci

Signed-off-by: Glenn Jocher <[email protected]>

glenn-jocher · 2022-09-04T11:21:49Z

@VELCpro thanks for the updates, this looks a lot simpler now! Is this ready to merge? Have you tested on videos?

VELCpro · 2022-09-04T12:37:01Z

@glenn-jocher yes I tested on videos and on a folder with multiple videos and seems ok.

I didn't test the piece of code about rtsp stream but seems trivial. The change only added a variable, the logic behind is untouched

Signed-off-by: Glenn Jocher <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Glenn Jocher <[email protected]>

glenn-jocher · 2022-09-04T14:17:27Z

@VELCpro looks good now. The only issue I see is that the default naming and value make it seem like we are skipping every 1 frame, i.e. defaulting to 15 FPS. Not sure if there's a better wording here like vid_stride or something. What do you think?

VELCpro · 2022-09-04T14:52:26Z

@glenn-jocher yes actually naming the variable skip_frame could cause some confusion.

vid_stride seems ok to me

glenn-jocher · 2022-09-04T14:58:37Z

@VELCpro perfect. I'll update.

glenn-jocher · 2022-09-04T15:16:01Z

@VELCpro PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

VELCpro · 2022-09-04T15:26:01Z

@glenn-jocher happy to help 🙂

* fps feature/skip frame added * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * predict.py updates * Update dataloaders.py Signed-off-by: Glenn Jocher <[email protected]> * Update dataloaders.py Signed-off-by: Glenn Jocher <[email protected]> * remove unused attribute Signed-off-by: Glenn Jocher <[email protected]> * Cleanup Signed-off-by: Glenn Jocher <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update predict.py Signed-off-by: Glenn Jocher <[email protected]> * Update detect.py Signed-off-by: Glenn Jocher <[email protected]> * Update dataloaders.py Signed-off-by: Glenn Jocher <[email protected]> * Rename skip_frame to vid_stride * cleanup * cleanup2 Signed-off-by: Glenn Jocher <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Glenn Jocher <[email protected]>

Mohammad-Saad-eng · 2023-08-18T10:35:50Z

@VELCpro @glenn-jocher Hello, hope you are doing well. I have trained a custom model on yolov8n which will detect the night traffic. I have to use it on 8 streams (or 4 in worst case) but I am facing too much lag. And when I use vid_stride, it skips the frames but the stream is still very slow. Is there a way to get better fps for that much streams using yolov8n?

VELCpro · 2023-08-18T15:07:37Z

Hi @Mohammad-Saad-eng everything good thank you for asking.
What about you?

As you notice the vid_stride parameter allows to skip frames during the detection over a video.
This could be useful if its not needed to do detection on every single frame in real time or in doing bulk analysis on large video files.

Other ways to improve latency without touching the actual models are for example to reduce the input size of the frames given to the model for the detection.

Clearly it's a matter of trade-offs between latency and accuracy.
It's probable that you are hitting the hardware boundaries of your system with the specific yolo model selected.

What hardware you are testing the model on?
Have you tested the model on a single stream to validate the latency?

Mohammad-Saad-eng · 2023-08-18T15:21:59Z

@VELCpro I am also fine.

I have tested it on Jetson Nano so far. Jetson is currently using DeepStream and TrafficCamNet for 8 streams. As number of parameters in yolov8n is significantly smaller than TrafficCamNet, I figured yolo would work fine too.

VELCpro · 2023-08-28T14:05:10Z

Hi @Mohammad-Saad-eng

Me and my team are currently using a Jetson orin 32GB in conjunction with yolov8n to perform object tracking obtaining good results both in terms of accuracy and fps.

The Jetson Nano should be capable of running yolov5n (despite the limited power and the different architecture in respect to yolov8n)

Here you can find an fps benchmark analysis performed by seeed studio where yolov5n6 run FP16 inferences in 48ms on a Jetson Nano.

I think also they used directly the yolo docker container without deepstream.
To further enhance speed other than using vid_stride you can try TesorRT model format in FP16 or INT8 to speed up the computation at inference time. Here a benchmark of the speed gains with different model format and jeston devices

Hope it helps

glenn-jocher · 2023-08-28T15:36:53Z

Hi @VELCpro-eng,

We have successfully used a Jetson Orin 32GB with yolov8n for object tracking, achieving good results in terms of both accuracy and FPS.

Although the Jetson Nano has limited power and a different architecture compared to yolov8n, it should still be capable of running yolov5n. Seeed Studio has performed an FPS benchmark analysis, running yolov5n6 in FP16 inferences in 48ms on a Jetson Nano. You can find the analysis here.

In addition to using the vid_stride parameter, you can also try using the TensorRT model format in FP16 or INT8 to further enhance speed during inference. Seeed Studio has conducted a benchmark testing the speed gains with different model formats on Jetson devices, which you can find here.

I hope these suggestions help.

github-actions bot reviewed Sep 2, 2022

View reviewed changes

VELCpro force-pushed the feature/skip_frame branch from 869c010 to 5288688 Compare September 2, 2022 16:20

fps feature/skip frame added

5f59213

VELCpro force-pushed the feature/skip_frame branch from 5288688 to 5f59213 Compare September 2, 2022 16:38

pre-commit-ci bot and others added 4 commits September 2, 2022 16:39

[pre-commit.ci] auto fixes from pre-commit.com hooks

6988627

for more information, see https://pre-commit.ci

predict.py updates

85d5123

Merge branch 'master' into feature/skip_frame

6fcdcdd

Update dataloaders.py

d6aa10c

Signed-off-by: Glenn Jocher <[email protected]>

VELCpro and others added 9 commits September 4, 2022 14:42

Merge branch 'ultralytics:master' into feature/skip_frame

c1295e6

Update dataloaders.py

25172f9

Signed-off-by: Glenn Jocher <[email protected]>

remove unused attribute

acf4e7c

Signed-off-by: Glenn Jocher <[email protected]>

Cleanup

cebdeb8

Signed-off-by: Glenn Jocher <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

7fe6acd

for more information, see https://pre-commit.ci

Update predict.py

2738688

Signed-off-by: Glenn Jocher <[email protected]>

Update detect.py

afb9575

Signed-off-by: Glenn Jocher <[email protected]>

Update dataloaders.py

365beec

Signed-off-by: Glenn Jocher <[email protected]>

Merge branch 'master' into feature/skip_frame

88f62d6

Merge branch 'master' into feature/skip_frame

7fc6bb3

Rename skip_frame to vid_stride

2b54f54

glenn-jocher changed the title ~~Select inference frame rate / skip frames~~ Add new --vid-stride inference parameter for videos Sep 4, 2022

glenn-jocher added 2 commits September 4, 2022 17:11

cleanup

29845e8

cleanup2

c9351e1

glenn-jocher merged commit 1aea74c into ultralytics:master Sep 4, 2022

VELCpro deleted the feature/skip_frame branch September 11, 2022 08:14

Hojland mentioned this pull request Oct 17, 2022

feat/bump Go-Autonomous/yolov5#15

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new `--vid-stride` inference parameter for videos #9256

Add new `--vid-stride` inference parameter for videos #9256

VELCpro commented Sep 2, 2022 •

edited by UltralyticsAssistant

Loading

github-actions bot left a comment

glenn-jocher commented Sep 2, 2022 •

edited

Loading

VELCpro commented Sep 2, 2022

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022 •

edited

Loading

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022

glenn-jocher commented Sep 4, 2022

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022

Mohammad-Saad-eng commented Aug 18, 2023

VELCpro commented Aug 18, 2023

Mohammad-Saad-eng commented Aug 18, 2023

VELCpro commented Aug 28, 2023 •

edited

Loading

glenn-jocher commented Aug 28, 2023

Add new --vid-stride inference parameter for videos #9256

Add new --vid-stride inference parameter for videos #9256

Conversation

VELCpro commented Sep 2, 2022 • edited by UltralyticsAssistant Loading

🛠️ PR Summary

🌟 Summary

📊 Key Changes

🎯 Purpose & Impact

github-actions bot left a comment

Choose a reason for hiding this comment

glenn-jocher commented Sep 2, 2022 • edited Loading

VELCpro commented Sep 2, 2022

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022 • edited Loading

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022

glenn-jocher commented Sep 4, 2022

glenn-jocher commented Sep 4, 2022

VELCpro commented Sep 4, 2022

Mohammad-Saad-eng commented Aug 18, 2023

VELCpro commented Aug 18, 2023

Mohammad-Saad-eng commented Aug 18, 2023

VELCpro commented Aug 28, 2023 • edited Loading

glenn-jocher commented Aug 28, 2023

Add new `--vid-stride` inference parameter for videos #9256

Add new `--vid-stride` inference parameter for videos #9256

VELCpro commented Sep 2, 2022 •

edited by UltralyticsAssistant

Loading

glenn-jocher commented Sep 2, 2022 •

edited

Loading

VELCpro commented Sep 4, 2022 •

edited

Loading

VELCpro commented Aug 28, 2023 •

edited

Loading